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1  INTRODUCTION 


Cirrus  clouds  have  been  shown  to  have  an  impact  on  certain  Air  Force  missions 
and  systems.  Some  examples  are  high  altitude  reconnaissance  and  laser  propagation 
for  communications  and  missile  defense.  Koenig  et  al.  (1993)  analyzed  the  potential 
impact  of  cirrus  on  laser  propagation  by  modeling  the  transmission  through  cirrus 
properties  assumed  for  distinct  altitude  layers.  Cirrus  extinction  was  derived  based  on 
the  assumption  of  spherical  ice  particles.  They  found  that  the  laser  power  loss  in  long 
path  lengths  depended  on  the  assumed  cirrus  extinction  when  the  laser  is  directed 
nearly  horizontally.  Norquist  et  al.  (2008)  executed  laser  transmission  models  in  slant 
paths  using  cirrus  cloud  properties  as  retrieved  from  observations.  Their  results  showed 
that  the  laser  beam  can  experience  significant  attenuation  in  long  paths  through  the 
cirrus  when  the  source  and  receiver  are  close  in  altitude  but  separated  by  distances  of 
30  km  or  more  horizontally.  Therefore,  accurate  predictions  of  cirrus  clouds  are  needed 
for  laser  system  mission  planning  in  advance  of  their  deployment. 

Having  characterized  the  effects  of  cirrus  on  laser  propagation  in  the  earlier 
study,  it  is  necessary  to  document  the  accuracy  of  cirrus  predictions  by  current  weather 
prediction  systems.  Norquist  et  al.  (2008)  state  that  since  any  visible  cirrus  clouds  are 
likely  to  have  an  appreciable  impact  on  laser  propagation,  their  macrophysical 
properties  (location,  altitude,  coverage)  are  probably  more  important  for  mission 
planning  than  their  microphysical  properties.  Thus,  the  current  study  focuses  on  the 
ability  of  mesoscale  weather  prediction  (MWP)  models  to  predict  the  spatial  and 
temporal  characteristics  of  cirrus  over  a  region  of  interest.  In  particular,  emphasis  is 
placed  on  determining  area  of  coverage  by  cirrus  and  the  reliability  of  MWP  models  to 
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correctly  predict  which  portions  of  the  region  will  be  covered.  Altitude  and  ice  water 
mass  of  the  cirrus  are  considered  but  are  of  secondary  importance  in  this  project. 

Some  recent  studies  have  been  conducted  to  evaluate  the  ability  of  MWP  models 
to  predict  ice  clouds.  Chiriaco  et  al.  (2006)  compared  the  predictions  by  several 
microphysical  schemes  in  the  fifth-generation  Pennsylvania  State  University-National 
Center  for  Atmospheric  Research  (PSU-NCAR)  Mesoscale  Model  (MM5;  Dudhia  1993) 
with  active  and  passive  remote  sensors  at  a  single  location  over  long  time  periods.  Their 
focus  was  on  the  microphysical  and  optical  properties  of  the  clouds,  and  they  found  that 
the  best  scheme  simulated  cirrus  at  the  site  in  about  2/3  of  the  selected  cirrus  events. 
Lewis  (2006)  looked  at  two  cirrus  episodes  in  the  southwestern  United  States  as 
simulated  by  the  MM5  as  executed  operationally  at  the  Air  Force  Weather  Agency 
(AFWA).  A  subjective  assessment  of  the  mass,  motion  and  moisture  fields  of  the  model 
predictions  showed  that  the  dynamical  and  hydrological  processes  necessary  for  cirrus 
formation  were  present,  but  that  for  the  two  cases  studied  they  produced  less  cirrus  in 
area  covered  than  was  detected  from  satellite  imagery. 

The  goal  of  this  project  was  to  comprehensively  assess  the  cirrus  prediction 
performance  of  the  AFWA  MM5  and  the  National  Center  for  Environmental  Prediction 
(NCEP)  North  American  Mesoscale  (NAM)  models.  Diagnoses  of  cirrus  clouds  from  the 
AFWA  Diagnostic  Cloud  Forecast  (DCF)  algorithm  (AFWA,  2007a)  applied  as  a  post 
processor  to  AFWA  MM5  forecasts  were  also  evaluated.  The  three  forecast  methods 
were  assessed  over  approximately  30-day  periods  in  each  season  over  two  different 
climatic  regions.  Satellite  imagery  was  used  as  a  reference  for  these  evaluations  to 
allow  assessments  of  the  models’  ability  to  predict  cirrus  on  horizontal  scales  of  100s  of 


2 


km.  This  assessment  will  provide  information  on  the  use  of  these  forecasts  to  support 
Department  of  Defense  system  tests  in  the  near  future. 

In  this  paper,  results  of  comparing  MM5  and  NAM  model  predictions  and  DCF 
algorithm  forecasts  of  ice  cloud  with  satellite  depictions  are  presented  for  periods  in 
Winter,  Spring  and  Summer  2007  over  two  regions  in  North  America:  the  northeast 
United  States  (NEUS)  region  (40-47  N  latitude,  70-81  W  longitude)  and  the  southwest 
United  States  (SWUS)  region  (30-40  N  latitude,  105-122  W  longitude).  Fall  2006  MM5 
and  NAM  predictions  were  also  assessed.  Twice-daily  24-  and  36-hour  forecasts  were 
compared  with  ice  cloud  masks,  ice  water  path  and  cirrus  top  height  as  retrieved  from 
the  Geostationary  Operational  Environmental  Satellite  -  12  (GOES-12)  imagery  for  the 
NEUS  region  and  GOES-1 1  imagery  for  the  SWUS  region.  Statistics  based  on  objective 
comparisons  of  the  gridded  representations  of  model  and  algorithm  forecasts  and 
GOES  analyses  are  reported  for  the  two  seasonal  periods  in  both  regions.  Some 
preliminary  conclusions  are  reached  about  the  characteristics  of  the  cirrus  predictions  of 
the  models/algorithms  based  on  this  study. 

2  DATA 

AFWA  executes  the  MM5  over  North  America  on  a  grid  of  45  km  grid  spacing 
with  a  nested  window  over  the  continental  United  States  (CONUS)  of  15  km  grid 
spacing.  Both  nests  are  discretized  vertically  into  41  model  layers  with  a  model  top  at  50 
hPa.  Physical  parameterizations  relevant  to  cirrus  formation  utilized  in  the  AFWA  MM5 
are  the  moist  convective  scheme  of  Kain  and  Fritsch  (1990)  and  the  explicit 
microphysics  formulation  of  Reisner  et  al.  (1998).  The  model  is  executed  at  06  and  18 
UTC  each  day  out  to  72  hours  (45  km  grid)  and  48  hours  (15  km  grid). 
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AFWA  MM5  06  and  18  UTC  15  km  grid  forecasts  of  21,  24,  33  and  36  hours 
were  obtained  each  day  for  the  periods  1  November  -  2  December  2006  (Fall),  13 
February  -  14  March  2007  (Winter),  8  May  -  6  June  2007  (Spring)  and  1-30  August 
2007  (Summer).  Pressure,  temperature,  height,  humidity  and  ice  water  mixing  ratio  (£ 
0.001  g/kg)  on  model  layers  were  extracted  for  the  NEUS  and  SWUS  regions.  These 
data  represented  the  MM5  forecasts  in  the  comparisons  conducted  in  this  project. 

NCEP  executes  the  Weather  Research  and  Forecasting  (WRF)  system  with  the 
Nonhydrostatic  Mesoscale  Model  (NMM)  core,  which  is  referred  to  as  the  WRF-NMM 
(NCEP,  2007).  NCEP  currently  utilizes  this  forecasting  system  as  their  North  American 
Mesoscale  (NAM)  weather  prediction  model.  Forecasts  of  12  km  grid  spacing  are 
executed  from  00,  06  12  and  18  UTC  daily.  NCEP  interpolates  the  forecast  fields  to  a 
Lambert  Conformal  grid  of  32.46  km  grid  spacing  covering  North  America  and 
surrounding  ocean  areas,  on  constant  pressure  levels  of  25  hPa  intervals  to  50  hPa, 
then  30,  20  and  10  hPa  levels.  The  current  operational  NAM  uses  the  Janjic  (1994) 
moist  convective  scheme  and  the  Ferrier  et  al.  (2002)  explicit  cloud  microphysics 
algorithm. 

NAM  06  and  18  UTC  forecasts  of  21,  24,  33  and  36  hours  duration  on  the  32.46 
km  grid  were  obtained  each  day  for  the  same  time  periods  as  mentioned  above  in  the 
AFWA  MM5  data  description.  Gridded  temperature,  height,  humidity  and  ice  water 
mixing  ratio  (>  0.001  g/kg)  on  constant  pressure  levels  were  extracted  for  the  NEUS  and 
SWUS  regions.  These  data  constituted  the  NAM  forecasts  utilized  in  the  comparisons 
for  this  report. 

AFWA’s  DCF  algorithm  is  executed  as  a  postprocessor  to  the  MM5  model  and 
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the  World-wide  Merged  Cloud  Analysis  (WWMCA,  AFWA,  2007b).  Ten  days  of  twice- 
daily  MM5  forecasts  are  paired  with  forecast  valid  time  WWMCA  gridded  analyses  to 
generate  statistical  relationships  between  selected  MM5  predictor  variables  and  the 
WWMCA  cloud  properties  (cloud  cover,  top  and  base  altitude  and  type)  for  as  many  as 
four  cloud  layers.  The  relationships  are  then  applied  to  ensuing  MM5  forecasts  to 
diagnose  cloud  layer  properties  at  prediction  times. 

AFWA  DCF  06  and  18  UTC  forecasts  of  21,  24,  33  and  36  hours  on  the  CONUS 
grid  of  15  km  spacing  were  acquired  each  day  for  the  time  periods  mentioned  above. 
Cloud  cover  (aka  amount),  cloud  top  height,  cloud  base  height  and  cloud  type  for  up  to 
three  layers  of  clouds  were  extracted  for  the  NEUS  and  SWUS  regions.  Only  cloud 
properties  for  diagnosed  clouds  (cloud  cover  diagnosed  to  be  greater  than  0%)  are 
included  in  the  data.  Zero  values  indicate  the  absence  of  clouds  in  some  or  all  of  data 
fields  corresponding  to  the  three  cloud  layers. 

GOES-11  imagery  channels  of  0.65,  3.7,  6.7  10.8  and  12  pm  on  picture  elements 
(pixels)  of  approximately  5  km  on  a  side  were  obtained  from  the  Naval  Research 
Laboratory  in  Monterey,  CA.  Image  files  containing  data  for  western  North  America  and 
the  eastern  Pacific  Ocean  were  obtained  for  image  times  of  06  and  18  UTC  (or  if 
missing,  the  nearest  available  image  times  prior  to  but  within  three  hours  of  these 
times).  Imagery  data  were  extracted  for  the  SWUS  region.  GOES-12  imagery  files 
created  by  the  Air  Force  Research  Laboratory  GOES-12  ground  station  at  image  times 
nearest  but  before  06  and  18  UTC  contained  data  for  eastern  North  America  and  the 
western  Atlantic.  GOES-12  imagery  channels  of  0.65,  3.7,  6.7  10.8  and  13  pm  on 
approximately  5  km  pixels  were  extracted  for  the  NEUS  region.  The  GOES-11  and 
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GOES-12  imagery  data  served  as  the  basis  for  comparison  with  MM5,  NAM  and  DCF 
ice  cloud  predictions  in  the  NEUS  and  SWUS  regions. 

3  METHOD 

Each  regional  GOES  image  file  was  processed  to  conduct  cloud  detection,  cloud 
phase  discrimination  and  cloud  property  retrieval.  The  cloud  detection  and  property 
retrieval  algorithm  of  Gustafson  and  d’Entremont  (2000)  was  utilized  to  distinguish 
between  clear,  liquid  cloud  and  ice  cloud  pixels,  and  for  the  ice  pixels  it  retrieved 
estimates  of  (among  other  properties)  ice  water  path  (IWP),  cloud  top  height  (CTH)  and 
visible  optical  depth.  AFWA  MM5  forecast  height  and  temperature  profiles  on  the  15  km 
grid  were  used  to  convert  retrievals  of  cloud  top  temperature  to  CTH.  Norquist  et  al. 
(2008)  compared  GOES-12  ice  cloud  retrievals  from  this  algorithm  with  radar  and  lidar 
measurements  of  CTH  and  radar/lidar  retrievals  of  IWP  for  a  series  of  cirrus  events  at 
Hanscom  Air  Force  Base,  MA  in  2005.  They  found  that  the  GOES-12  CTH  estimates 
were  on  average  about  1.1  km  lower  in  altitude  than  the  radar/lidar  measurements. 
They  also  found  that  GOES-12  IWP  retrievals  were  on  average  smaller  than  the 
radar/lidar  retrievals  by  a  factor  of  3  to  4.  Cloud  detection  is  considered  the  strongest 
feature  of  the  algorithm,  and  cloud  phase  discrimination  is  also  thought  to  be  reliable.  It 
should  be  emphasized,  however,  that  Norquist  et  al.  (2008)  found  that  the  GOES  ice 
cloud  detection  failed  to  detect  cirrus  with  visible  optical  depths  of  less  than  0.5  as 
determined  from  the  radar/lidar  retrievals.  Therefore,  the  cirrus  detected  by  the  GOES 
algorithm  are  likely  limited  to  an  optical  depth  of  about  0.5  and  greater  and  might  not 
account  for  optically  thinner  cirrus.  This  may  result  in  a  slight  under  representation  of 
the  total  cirrus  coverage,  especially  near  the  cloud  edges. 
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The  first  step  in  the  forecast  model  -  GOES  comparison  processing  was  to 
identify  the  GOES  pixels  associated  with  (lying  closest  to)  each  MM5,  NAM  or  DCF  grid 
point.  These  pixels  can  be  thought  of  as  lying  within  the  corresponding  model/algorithm 
grid  cell.  For  the  15  km  MM5  and  DCF  grids,  on  average  9  pixels  were  within  each  cell, 
while  for  the  32.5  km  NAM  grid,  about  40  pixels  on  average  were  within  each  cell.  This 
needed  to  be  done  just  once  for  each  combination  of  region  and  forecast  model. 

MM5  and  NAM  forecast  fields  of  temperature,  height,  humidity  and  ice  water 
mixing  ratio  (and  for  MM5,  pressure)  were  used  in  the  processing  to  compare  ice  cloud 
predictions  with  satellite  imagery.  In  each  model  grid  cell,  the  tropopause  is  computed 
from  the  pressure,  temperature  and  height  profile  using  the  algorithm  of  Roe  and 
Jasperson  (1980).  Vertical  profiles  of  predicted  pressure  (for  MM5  only),  temperature, 
height,  humidity  and  ice  water  content  (converted  from  ice  water  mixing  ratio  by 
multiplying  by  air  density),  and  the  computed  tropopause,  were  interpolated  in  time  (if 
necessary)  between  the  21  and  24  or  33  and  36  hour  forecast  times,  to  the  time  of  the 
comparative  GOES  image.  For  the  time-interpolated  MM5  profiles,  the  height  at  the  top 
of  each  model  layer  was  computed.  The  average  height  between  constant  pressure 
levels  was  used  as  the  height  at  the  top  of  each  “pressure  layer”  for  NAM.  Next,  the 
model  layers  (or  pressure  layers  for  NAM)  with  non-zero  ice  water  content  were 
identified,  and  were  numerically  integrated  (each  multiplied  by  layer  depth  and  summed) 
to  compute  the  IWP,  and  the  height  of  the  top  of  the  topmost  ice  cloud  layer  was  set  as 
the  CTH.  Only  model  grid  cells  with  CTH  >  6  km  were  considered  in  the  comparison 
with  GOES.  Such  a  grid  cell  was  counted  as  a  model  ice  grid  point  and  it’s  computed 
IWP  and  CTH  were  stored. 
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In  the  DCF  processing,  three  layers  of  the  following  four  forecast  quantities  were 
used  from  the  pair  of  forecast  times  bracketing  the  imagery  time:  cloud  cover,  cloud 
type,  cloud  base  altitude  and  cloud  top  altitude.  It  also  used  all  of  the  MM5  model  levels 
of  pressure,  temperature  and  height  from  the  two  corresponding  MM5  files,  as  well  as 
the  MM5  grid  point  terrain  height  values.  The  MM5  quantities  were  used  to  compute  the 
tropopause  height  following  the  method  of  Roe  and  Jasperson  (1980)  for  each  regional 
grid  point  for  the  two  MM5  times.  The  MM5  grid  point  lying  closest  to  each  DCF  grid 
point  provided  the  tropopause  height  for  use  in  the  DCF  processing.  The  algorithm  then 
processed  each  within-region  DCF  grid  point  in  turn.  The  algorithm  considered  each  of 
the  three  layers  of  the  time-pairs  of  cloud  cover,  type,  base  and  top  values.  For  each  of 
the  two  forecast  times,  it  identified  (if  any)  the  highest  (in  altitude)  of  the  three  cloud 
layers  with  non-zero  cloud  cover  and  with  a  cirrus  type  (types  8  or  9)  that  had  a  top 
height  £  6  km.  If  only  one  of  the  two  forecast  times  had  such  a  layer,  the  cover  was  time 
interpolated  (to  the  imagery  time)  using  a  zero  value  at  the  appropriate  forecast  time. 
The  nonzero  forecast  time’s  top  height  was  assigned  as  the  interpolated  top  height.  If 
both  forecast  times  have  such  a  layer,  a  linear  interpolation  in  time  to  the  imagery  time 
was  used  for  both  the  cover  and  top  height.  The  nearest  neighbor  MM5  tropopause 
heights  were  also  interpolated  in  time  to  the  imagery  time.  At  this  point  the  processing  of 
interpolated  DCF  cirrus  cover,  top  height  and  MM5  tropopause  height  proceeded  in  a 
manner  analogous  to  that  for  the  MM5  and  NAM  cirrus  ice  water  path,  top  height  and 
tropopause  height  grid  values. 

GOES  ice  pixels  within  the  model/algorithm  grid  cell  with  a  CTH  >  6  km  (if  any) 
were  counted,  and  for  a  non-zero  count  the  retrieved  IWP  and  CTH  were  averaged.  The 
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fraction  of  ice  pixels  within  the  grid  cell  represented  the  percent  GOES  cirrus  cover.  If 
the  within-cell  ice  pixel  count  was  at  least  one,  then  the  collective  within-cell  ice  pixels 
were  considered  as  a  GOES  ice  grid  point  and  the  averaged  IWP  and  CTH,  along  with 
the  cover,  were  stored.  If  both  model/algorithm  and  GOES  were  ice  grid  points,  then  the 
fcst  Y  /  GOES  Y  counter  was  incremented  for  that  grid  cell  and  the  respective  IWP 
(cover  for  DCF)  and  CTH  values  were  paired  for  comparison.  If  either  or  both  model  or 
GOES  were  not  ice  grid  points,  then  the  corresponding  category  (fcst  Y  /  GOES  N,  fcst 
N  /  GOES  Y,  fcst  N  /  GOES  N)  was  incremented. 

There  are  two  reasons  why  a  single  GOES  pixel  is  considered  sufficient  to 
identify  the  model  grid  point  as  an  ice  grid  cell  for  the  purposes  of  satellite  detection 
First,  as  previously  mentioned  the  GOES  ice  cloud  detection  is  limited  to  visible  optical 
depths  of  about  0.5  and  higher.  The  presence  of  any  detected  ice  pixels  in  a  grid 
suggests  the  likelihood  of  undetected  ice  pixels  nearby.  Second,  it  is  important  to 
account  for  the  presence  of  any  cirrus  in  evaluating  the  predictive  models/algorithm. 
Because  of  the  need  to  avoid  cirrus  in  certain  operations,  it  is  important  that  the 
prediction  systems  anticipate  any  cirrus  occurrence  in  a  region  of  interest. 

Once  all  model/algorithm  grid  cells  were  processed  for  an  individual  forecast  time 
(24-h  or  36-h),  the  total  number  of  all  model/algorithm  ice  grid  points,  the  total  number 
of  all  GOES  ice  pixels,  and  the  count  of  the  four  ice  grid  point  Y/N  categories  were 
recorded,  along  with  the  IWC,  cover  and  CTH  pairs  and  tropopause  height  for  the  Y/Y 
ice  grid  points.  Then  after  processing  of  all  forecasts  for  both  24-h  and  36-h  durations 
was  completed  for  the  seasonal  period,  a  totaling  of  the  individual  forecast  totals  and 
their  percentages  of  the  total  number  of  grid  points  (for  total  model/algorithm  ice  grid 
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points  and  Y/N  category  counts)  and  total  number  of  GOES  pixels  was  computed. 
Individual  forecast  averages  and  overall  period  averages  of  the  models/algorithm  and 
GOES  IWP,  cover  and  CTH  was  computed  for  all  Y/Y  ice  grid  points.  In  addition,  all 
such  CTH  values  were  compared  to  the  tropopause  heights  for  each  grid  point,  and  if  it 
exceeded  the  trop  height  it  was  adjusted  to  the  trop  height.  Then  a  subsequent  CTH 
average  was  computed  over  all  Y/Y  ice  grid  points  for  the  models,  algorithm  and  GOES, 
which  is  referred  to  as  the  trop  adjusted  CTH  averages.  Finally  a  count  and  percentage 
of  the  Y/Y  grid  points  whose  CTH  was  so  adjusted  were  computed. 

4  RESULTS 

Regional  cirrus  coverage  and  location  of  cirrus  within  the  region  are  the  primary 
metrics  of  interest  in  the  current  prediction  evaluation.  The  assessment  of  the 
model/algorithm  forecasts  begins  with  their  performance  on  the  regional  scale,  and  then 
investigates  the  placement  of  predicted  cirrus  within  the  regions.  First,  examples  of 
selected  forecasts  are  illustrated  as  forecast  maps  and  compared  with  maps  of  detected 
cirrus.  Next,  summary  statistics  of  the  24-  and  36-hour  ice  cloud  forecasts  and  GOES 
ice  cloud  detection  over  the  four  seasonal  periods  are  compared.  This  is  followed  by  a 
review  of  the  fcst/GOES  Y/N  category  totals  over  each  seasonal  period,  an  indicator  of 
overall  forecast  skill.  Indices  drawn  from  the  Y/N  statistics  are  discussed.  Finally,  the 
predicted  and  retrieved  ice  water  path,  cover  and  cirrus  top  height  at  the  fcst  Y  /  GOES 
Y  grid  cells  are  compared. 

4.1  Selected  Map  Comparisons 

Cirrus  forecast  maps  from  the  MM5  and  NAM  models  and  the  DCF  algorithm  are 
shown  in  Figures  1-19  in  comparison  with  maps  of  retrieved  GOES  cirrus  properties. 
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MM  5  21  6500  HR  CIRRUS  CTH  FCS1  VALiU  0359  UtC  20061102  MAM  21.6500  HR  CIRRUS  C1H  1-CSI  VALID  0339  UTC  20061102 


Figure  1 .  NEUS  cirrus  top  height  (km)  valid  at  0339  UTC  2  November  2006  from  21  65-h  MM5,  NAM  forecasts  and 
GOES- 12  analysis.  Gray  scales  intervals  are  1  km  starting  with  black  shading  at  6  km. 


MMb  21.6500  HR  CIRRUS  IWP  h'CST  VALID  0559  UVC  20061  102  NAM  21.6500  HR  CIRRUS  IWP  FCS1  VALID  0559  UTC  2C061  102 


Figure  2.  Same  as  in  Figure  1  except  ice  water  path  (g  m'2).  Gray  scale  interval  is  25  g  m'2  starting  with  black  at  25  g  m 2. 


MM5  36.0000  HR  CIRRUS  CTH  FCST  VALID  0600  UTC  2006  i  103  NAM  36.0000  HR  CIRRUS  C1H  FCST  VALID  0600  UTC  20061  103 


Figure  3.  SWUS  cirrus  top  height  (km)  valid  at  0600  UTC  3  November  2006  from  36-h  MM5,  NAM  forecasts  and  GOES- 
1 1  analysis  as  labeled.  Gray  scale  intevals  are  1  km  starting  with  black  shading  at  6  km. 


MM3  36.0000  HR  CIRRUS  iWH  HCS1  VAUO  0600  UlC  2U061  103  NAM  36.0000  HR  CIRRUS  IWP  COST  VAl.lD  0600  UTC  20061103 


Figure  4.  Same  as  in  Figure  3  except  ice  water  path  (g  m'z).  Gray  scale  interval  is  25  g  m 2  starting  with  black  at  25  g  m 2. 


MM5  21.6500  HR  CIRRUS  CIH  FCST  VALID  0339  UIC  200/0214  NAM  21.6500  HR  CIRRUS  CfH  FCST  VALID  0339  UTC  20070214 


. 


Figure  5.  NEUS  cirrus  top  height  (km)  valid  0339  UTC  14  February  2007  from  21.65-h  MM5,  NAM,  DCF  forecasts  and 
GOES-12  analysis  as  labeled.  Contour  and  gray  scale  intervals  are  1  km. 


MMb  21.6500  ri H  CIRRUS  iWP  FCST  VALID  0339  UIC  200^0214  NAM  21,6500  HR  CIRRUS  IWP  FCST  VAUD  0339  UTC  20070214 


Figure  6.  Same  as  in  Figure  5  except  ice  water  path  (g  m'2).  Contour  and  gray  scale  intervals  are  2"  g  m'2,  n=0,  9. 


DCF  21.6500  HR  CIRRUS  CC  FCST  VAtiD  05.39  UTC  200/0214  GOES  CIRRUS  CC  RLIRlEVAL  VALID  0559  UTC  200702  M 


Figure  7.  Cirrus  cover  (%)  from  NEUS  21.65-h  DCF  forecast  and  GOES-12  analysis  valid  0339  UTC  14  February  2007, 
and  from  SWUS  24-h  DCF  forecast  and  GOES-11  analysis  valid  1800  UTC  22  February  2007.  Contour  interval  is  10%. 


MM5  24.0000  HR  CIRRUS  CfH  TCSI  VALIO  1800  UK  20070222  NAM  24.0000  HR  CIRRUS  CTh  ICSI  VAl.10  1800  UTC  20070222 


Figure  8.  SWUS  cirrus  top  height  (km)  valid  1800  UTC  22  February  2007  from  24-h  MM5,  NAM,  DCF  forecasts  and 
GOES-1 1  analysis  as  labeled.  Contour  and  gray  scale  intervals  are  1  km. 


MMS  24.00U0  HR  CIRRUS  (WP  FCST  VALID  1800  UlC  20070222  MAM  24.0000  HR  CIRRUS  iWP  FCST  VALID  1800  UTC  20070222 


Figure  9.  Same  as  in  Figure  8  except  ice  water  path  (g  m'2).  Contour  and  gray  scale  intervals  are  2n  g  m 2,  n=0,  9. 


MMi  33.6MX)  HR  CiRKU'j  Cl h  rCSI  VALID  OJ JV  UK.  20U/0b05(  _ NAM  33.6500  HR  CIRRUS  CTH  FC5L  VALID  0339  UTC  200/0509 


Figure  10.  NEUS  cirrus  top  height  (km)  valid  0339  UTC  9  May  2007  from  35.65-h  MM5,  NAM.  DCF  forecasts  and  GOES- 
12  analysis  as  labeled.  Contour  and  gray  scale  intervals  are  1  km. 


MM5  33.6500  HR  CIRRUS  IWP  FCST  VALID  0339  UTC  20070509  NAM  33.6500  HR  CIRRUS  iWP  FCST  VALID  0339  UTC  20070509 


Figure  11.  Same  as  in  Figure  10  except  ice  water  path  (g  m'2).  Contour  and  gray  scale  intervals  are  2"  g  m'2,  n=0,  9. 


DCI-  33.6500  HR  CIRRUS  CC  FCS1  VALIU  03.59  UlC  200/0909  GOLS  CikftUS  CC  RETRIEVAL  VALID  0339  UTC;  200/0509 


Figure  12.  Cirrus  cloud  cover  {%)  from  NEUS  35.65-h  DCF  forecast  and  GOES-12  analysis  valid  0339  UTC  9  May  2007, 
and  from  SWUS  24-h  DCF  forecast  and  GOES-11  analysis  valid  0600  UTC  8  May  2007.  Contour  interval  is  10%. 


MM5  24-  0000  HR  CIRRUS  CTH  FCSI  VALID  (J6C0  UfC  20070508  NAM  24.0000  HR  CIRRUS  CTH  FCSI  VALID  0600  UTC  20070508 


analysis  as  labeled.  Contour  and  gray  scale  interval  is  1  km. 


MM5  24.0000  HR  CIRRUS  IWP  f-CS<  VAf  ID  U60G  UTC  20070b0h  NAM  / 4  0000  HR  CIRRUS  1WP  FCS1  VALID  0600  UTC  2007Q5Q8 


Figure  14.  Same  as  Figure  13  except  ice  water  path  (g  m'2}.  Contour  and  gray  scale  interval  is  2n  g  m 2,  n-0,  9. 


MMb  3b.6S00  HR  CIRRUS  CIH  KCST  VALID  1 /39  UlL.  200/0821  NAM  3S.6500  HR  CIRRUS  ClH  KCS1  VALID  1/39  UTC  20070821 


Figure  15.  NEUS  cirrus  top  height  (km)  valid  1739  UTC  21  August  2007  from  35.65-h  MM5,  NAM.  DCF  forecasts  and 
GOES-12  analysis  as  labeled  Contour  and  gray  scale  interval  is  1  km. 


Figure  16.  NEUS  ice  water  path  (g  m'2)  valid  1739  UTC  21  August  2007  from  35.65-h  MM5,  NAM  forecasts  and  GOES-12 
analysis.  Lower  right  is  NAM  forecast  of  cloud  ice  mixing  ratio  plus  snow  water  mixing  ratio  converted  to  ice  water  path. 


DCF  3b. 6S00  HR  CIRRUS  CC  FCSt  VALID  1739  UTC  20070B21  GOES  CIRRUS  CC  RETRIEVAL  VALID  1739  UTC  20070821 


Figure  17.  Cirrus  cloud  cover  (%)  from  NEUS  35.65-h  DCF  forecast  and  GOES-12  analysis  valid  1739  UTC  21  August 
2007,  and  from  SWUS  36-h  DCF  forecast  and  GOES-11  analysis  valid  0600  UTC  3  August  2007.  Contour  interval  is  10%. 


MMb  ib.oooa  HR  CIRRUS  ClH  FC$T  VALID  U6UU  UtL  2Q0/0EU3  NAM  3b, GOOD  HR  CIRRUS  ClH  FCSf  VALID  060Q  UlC  200/080.3 


Figure  18.  SWUS  cirrus  top  height  (km)  valid  0600  UTC  3  August  2007  from  36-h  MM5,  NAM,  DCF  forecasts  and  GOES- 
1 1  analysis  as  labeled.  Contour  and  gray  scale  interval  is  1  km. 


MM5  J6.0000  HR  CIRRUS  iWP  FCST  VAliD  0600  UTC  20070ESUJ  NAM  jG.OOOO  HR  CIRRUS  iWP  FCST  VALID  0600  UTC  20070803 


Figure  19  SWUS  ice  water  path  (g  m  2)  valid  0600  UTC  3  August  2007  from  36-h  MM5,  NAM  forecasts  and  GOES-1 1 
analysis.  Lower  right  is  NAM  forecast  of  cloud  ice  mixing  ratio  plus  snow  water  mixing  ratio  converted  to  ice  water  path. 


These  depictions  are  shown  as  examples  of  selected  forecasts  and  are  not  intended  as 
a  means  of  objectively  assessing  forecast  performance. 

Some  issues  arise  from  these  figures  that  are  worth  noting.  First,  there  is  much 
more  spatial  scatter  and  irregularity  in  the  GOES  depictions  that  in  the  predictions. 
While  this  is  to  be  expected  to  some  degree  because  of  the  greater  spatial  resolution  of 
the  GOES,  it  should  be  recognized  that  the  predictive  models/algorithm  cannot 
reproduce  all  of  the  fine  detail  of  the  cirrus  apparent  in  nature.  The  most  that  can  be 
expected  of  the  modeling  techniques  is  that  they  create  ice  clouds  generally  in  the 
locations  where  the  greatest  concentrations  of  cirrus  actually  occur.  Second,  it  is  clear 
in  many  cases  that  the  extent  of  area  coverage  is  less  in  IWP  depictions  than  is 
indicated  by  CTH.  A  good  example  of  this  can  be  seen  in  the  comparisons  of  the  MM5 
maps  in  Figures  10  and  1 1,  in  which  CTH  indicators  of  cirrus  presence  depict  a  band  of 
cirrus  across  the  southern  portion  of  the  region  that  does  not  appear  in  the  IWP  map. 
This  is  because  the  MM5  IWP  values  for  this  band  fell  below  1  g  m~2  and  did  not  get 
plotted  on  the  IWP  map.  The  GOES  algorithm  did  not  detect  the  southern  cirrus  band, 
but  because  we  know  that  very  thin  cirrus  are  often  not  detected  by  the  GOES 
algorithm,  we  can’t  say  for  sure  that  there  are  actually  no  cirrus  present  in  the  southern 
part  of  the  region.  This  suggests  some  level  of  uncertainty  in  the  evaluation  of  the  cirrus 
predictions  due  to  the  limitation  of  the  reference  that  is  being  used,  the  GOES  detection 
process.  Because  the  satellite  algorithm  can  under-detect  cirrus  but  not  over-detect  it,  a 
model/algorithm  under-prediction  is  of  greater  concern  than  an  over-prediction. 

4.2  Seasonal  Period  Summary  Statistics 

Tables  1-4  show  summary  statistics  for  the  model/algorithm  -  GOES 
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comparisons  for  both  regions  and  both  24-  and  36-hour  forecast  durations  in  the  Fall, 
Winter,  Spring  and  Summer  periods  respectively.  The  much  larger  SWUS  has  many 
more  model  grid  points  and  GOES  pixels  than  does  the  smaller  NEUS  region.  Relative 
sizes  of  the  MM5  and  DCF  (15  km)  and  NAM  (32.5  km)  grid  cells  results  in  more  than 


four  times  as  many  MM5  and  DCF  grid  points  than  NAM  points  in  each  region. 

Table  1.  Fall  period  summary  statistics  for  comparison  of  MM5  and  NAM  ice  cloud 
predictions  (average  duration  in  hours  in  parentheses)  with  GOES  ice  cloud  retrievals. 
Numbers  of  grid  points  and  pixels  shown  are  the  total  over  all  forecasts.  The  last  two 
columns  are  the  percentages  of  the  total  number  of  grid  points  or  pixels  in  which  ice 


Region/Fcst 

Duration 

Model 

#  Fcsts 

#  Model 
Grid  Pts. 

#  GOES 
Pixels 

%  Model 
Ice  Pts. 

%  GOES  1 
Ice  Pixels 

NEUS/24 

(22.7) 

MM5 

~29l 

83259 

802459 

541 

29 

NAM 

25 

16675 

694000 

48 

27 

NEUS/36 

(34.7) 

MM5 

29l 

83259 

802459 

48 

29 

NAM 

26 

17342 

721760 

46 

28 

SWUS/24 

(24.0) 

MM5 

~62 

458738 

4306892 

26 

28 

NAM 

58 

101094 

4018936 

26 

29 

SWUS/36 

(36.0) 

MM5 

62 

458738 

4306892 

27 

27 

NAM 

58 

101094 

4018936 

24 

29 

Table  2.  Winter  period  summary  statistics  for  comparison  of  MM5,  NAM  and  DCF  ice 
cloud  predictions  (average  duration  in  hours  in  parentheses)  with  GOES  ice  cloud 
retrievals.  Numbers  of  grid  points  and  pixels  shown  are  the  total  over  all  forecasts.  The 
last  two  columns  are  the  percentages  of  the  total  number  of  grid  points  or  pixels  in 
which  ice  cloud  was  predicted  or  detected  respectively. 


Region/Fcst 

Duration 

Model 

#  Fcsts 

#  Model 
Grid  Pts. 

#  GOES 
Pixels 

%  Model 
Ice  Pts. 

%  GOES 
Ice  Pixels 

NEUS/24 

(22.6) 

MM5 

53 

152163 

1466563 

401 

21 

NAM 

53 

35351 

1471280 

39 

21 

DCF 

49 

140679 

1355879 

16 

22 

NEUS/36 
(34  6) 

MM5 

52 

149292 

1438892 

40 

20 

NAM 

52 

34684 

1443520 

37 

20 

DCF 

47 

134937 

1300537 

14 

21 

SWUS/24 

(24.0) 

MM5 

59 

436541 

4098494 

26 

25 

NAM 

59 

102837 

4088228 

20 

25 

DCF 

54 

398304 

3739878 

7 

27 

SWUS/36 

(36.0) 

MM5 

58 

429142 

4029028 

26 

25 

NAM 

58 

101094 

4018936 

19 

25 

DCF 

51 

376176 

3532107 

4 

26 

31 


Table  3.  Same  as  in  Table  2  except  for  the  Spring 


period. 


Region/Fcst 

Duration 

Model 

#  Fcsts 

#  Model 
Grid  Pts. 

#  GOES 
Pixels 

%  Model 
Ice  Pts. 

%  GOES 
Ice  Pixels 

NEUS/24 

(22.7) 

MM5 

54] 

155034 

1494234 

42 

33 

NAM 

54 

36018 

1499040 

42 

32 

DCF 

48 

137808 

1328208 

29 

32 

NEUS/36 

(34.7) 

MM5 

54 

155034 

1494234 

43 

31 

NAM 

53 

35351 

1471280 

42 

311 

DCF 

47 

134937 

1300537 

29 

~3V 

SWUS/24 

(24.0) 

MM5 

49 

362551 

3403834 

26 

21 

NAM 

50 

87150 

3464600 

18 

20 

DCF 

49 

361424 

3393593 

1 

21 

SWUS/36 

(36.0) 

MM5 

49 

362551 

3403834 

27 

~2V 

NAM 

48 

83664 

3326016 

18 

21 

DCF 

49 

361424 

3393593 

1 

21 

Table  4,  Same  as  in  Table  2  except  for  the  Summer  period. 


Region/Fcst 

Duration 

Model 

#  Fcsts 

#  Model 
Grid  Pts. 

#  GOES 
Pixels 

%  Model 
Ice  Pts. 

%  GOES 
Ice  Pixels 

NEUS/24 

(22.6) 

MM5 

55 

157905 

1521905 

36 

26 

NAM 

53 

35351 

1471280 

37 

27 

DCF 

50 

143550 

1383550 

27 

27 

NEUS/36 

(34.6) 

MM5 

55 

157905 

1521905 

34 

26 

NAM 

~ 54 

36018 

1499040 

35 

26 

DCF 

51 

146421 

1411221 

26 

27 

SWUS/24 

(24.0) 

MM5 

57 

421743 

3959562 

16 

~2 2 

NAM 

55 

95865 

3811060 

15 

21 

DCF 

51 

376176 

3532107 

7 

22 

SWUS/36 

(36.0) 

MM5 

57 

421743 

3959562 

13 

22 

NAM 

55 

95865 

3811060 

14 

22 

DCF 

51 

376176 

3532107 

6 

22 

The  last  two  columns  of  Tables  1-4  compare  the  percentages  of  all 
model/algorithm  grid  points  (GOES  pixels)  for  which  ice  cloud  with  top  height  >  6  km 
was  predicted  (detected).  This  is  the  same  as  the  percentage  of  the  region  covered  by 
cirrus,  averaged  over  all  comparison  times.  GOES  cirrus  coverage  in  the  NEUS  is 
greater  in  the  Spring  and  Summer  periods  than  in  the  Winter,  whereas  the  opposite  is 
true  of  the  SWUS  region.  It  is  not  known  if  this  is  a  usual  occurrence  or  just  a 
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characteristic  of  the  single  year  studied.  In  any  case,  the  models  are  unable  to  replicate 
the  relative  regional  coverage  of  the  seasons  and  regions.  NEUS  cirrus  coverage  is 
over-predicted  on  average  by  both  MM5  and  NAM  in  both  seasons,  especially  in  the 
Fall  and  Winter  periods.  In  the  SWUS,  both  models  are  in  good  agreement  with  GOES 
cirrus  coverage  in  the  Fall  as  is  MM5  in  the  Winter.  MM5  over-predicts  in  the  Spring, 
whereas  NAM  under-predicts  in  the  Winter  and  Spring  and  both  models  produce 
significantly  less  cirrus  than  detected  by  GOES  in  the  Summer.  DCF  cirrus  coverage  in 
the  NEUS  is  biased  low  in  the  Winter  period,  slightly  low  in  the  Spring  but  about  right  in 
the  Summer  term.  DCF  SWUS  cirrus  coverage  is  much  less  than  GOES  coverage  in 
Winter,  Spring  and  Summer.  It  is  not  clear  what  happened  with  the  DCF  in  the  Spring 
over  the  SWUS  where  on  average  only  a  1%  regional  coverage  of  cirrus  was  predicted. 
Only  in  the  Winter  and  Spring  periods  did  the  MM5  predict  on  average  greater  regional 
cirrus  cover  than  did  NAM  in  the  SWUS.  There  are  no  such  clear  trends  between  the 
models  in  the  NEUS  regional  coverage  prediction,  except  perhaps  for  a  slightly  greater 
prediction  for  MM5  than  NAM  in  the  Fall  and  Winter. 

4.3  Regional  Cirrus  Coverage  by  Individual  Forecasts 

Prediction  of  the  timing  of  regional  cirrus  coverage  is  of  great  interest  to  mission 
planners  in  providing  guidance  for  system  deployment  decisions.  Figures  20-27  show 
the  percent  cirrus  coverage  of  the  NEUS  and  SWUS  regions  from  individual  forecasts 
by  the  models  and  algorithm  (24-h  forecasts)  and  the  GOES  analysis  for  the  Fall  - 
Summer  periods  in  turn.  Gaps  appearing  in  the  line  plots  shown  are  due  to  missing 
predictions,  verifying  time  GOES  analyses,  or  both.  Dash  markers  are  included  in  some 
of  the  plots  to  indicate  the  coverage  values  when  gaps  prevent  connecting  lines. 
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The  major  value  of  merit  in  the  individual  forecast  regional  cirrus  coverages  is  the 
ability  to  correctly  predict  the  maxima  and  minima  as  depicted  by  the  GOES  coverages. 
The  models  and  algorithm  show  a  varying  degree  of  this  ability.  MM5  and  NAM 
anticipate  virtually  all  of  the  GOES  maxima  and  minima  in  both  NEUS  and  SWUS  in  the 
Fall  period  (Figures  20  and  21).  However,  they  exaggerate  the  coverage  in  the  NEUS 
maxima  but  show  a  varying  degree  of  agreement  with  GOES  in  the  SWUS.  In  fact,  it 
appears  that  MM5  and  NAM  are  able  to  capture  virtually  all  of  the  maxima  cirrus 
regional  coverages  and  at  the  same  time  realistic  portray  their  durations.  Generally 
there  is  a  tendency  by  MM5  and  NAM  to  over-predict  the  maxima  in  the  NEUS  and 
either  correctly  predict  or  under-predict  in  the  SWUS.  DCF  performs  considerably  better 
in  the  NEUS  than  in  the  SWUS,  but  still  misses  a  few  maxima  in  the  former  region. 

4.4  Cirrus  Location  Forecast  Accuracy 

To  determine  the  optimum  locus  of  operations  in  a  region,  it  is  important  to  know 

in  advance  where  the  cirrus  clouds  will  be.  Percentages  of  grid  points  in  which  cirrus 

Y/N  presence  is  correctly  and  incorrectly  predicted  by  the  models/algorithms  are 

presented  in  contingency  tables.  Tables  5-8  are  the  contingency  tables  for  the  ice  cloud 

grid  point  Y/N  category  percentages  in  the  Fall,  Winter,  Spring  and  Summer  periods. 

Table  5.  Fall  period  contingency  table  (%)  for  comparison  of  grid  point  cirrus  forecasts 
and  GOES  within-grid  cell  cirrus.  GOES  =  Y  where  ice  cloud  was  detected  in  at  least 


GOES 


GOES 


MM5 

Y 

N 

NAM 

Y 

N 

NEUS/24 

Y 

32 

22 

Y 

33 

15 

N 

6 

40 

N 

12 

40 

NEUS/36 

Y 

28 

20 

Y 

31 

15 

N 

9 

43 

N 

14 

40 

SWUS/24 

Y 

18 

8 

Y 

19 

7 

N 

19 

55 

N 

27 

47 

SWUS/36 

Y 

17 

10 

Y 

18 

6 

N 

19 

54 

N 

28 

48 

42 


Table  6.  Same  as  Table  5  except  for  Winter  period. 

GOES  GOES 


GOES 


MM5 

Y 

N 

NAM 

Y 

N 

DCF 

Y 

N 

NEUS/24 

Y 

23 

17 

Y 

26 

13 

Y 

11 

4 

N 

5 

55 

N 

8 

53 

N 

18 

67 

NEUS/36 

Y 

21 

20 

Y 

24 

14 

Y 

9 

5 

N 

6 

54 

N 

9 

53 

N 

19 

67 

SWUS/24 

Y 

17 

9 

Y 

16 

3 

Y 

‘6 

1 

N 

17 

57 

N 

27 

54 

N 

31 

62 

SWUS/36 

Y 

16 

9 

Y 

14 

4 

Y 

4 

1 

N 

18 

57 

N 

28 

54 

N 

32 

63 

Table  7.  Same  as  Table  5  except  for  Spring  period. 

GOES  GOES  GOES 


MM  5 

Y 

N 

NAM 

Y 

N 

DCF 

Y 

N  n 

NEUS/24 

Y 

28 

14 

Y 

34 

8 

Y 

19 

10 

N 

14 

44 

N 

16 

42 

N 

22 

49 

NEUS/36 

Y 

28 

15 

Y 

32 

10 

Y 

19 

10  1 

N 

13 

44 

N 

17 

41 

N 

21 

50 

SWUS/24 

Y 

15 

11 

Y 

13 

5 

Y 

1 

0 

N 

13 

61 

N 

21 

61 

N 

27 

72 

SWUS/36 

Y 

15 

12 

Y 

13 

6 

Y 

1 

0 

N 

13 

60 

N 

21 

60 

N 

27 

72 

Table  8.  Same  as  Table  5  e 

GC 

xcept  for  Summer  period. 
)ES  GC 

IES  GOES 

MM5 

Y 

N 

NAM 

Y 

N 

DCF 

Y 

N 

NEUS/24 

Y 

22 

14 

Y 

27 

11 

Y 

17 

10 

N 

13 

51 

N 

17 

45 

N 

19 

54 

NEUS/36 

Y 

20 

14 

Y 

24 

11 

Y 

16 

10 

N 

15 

51 

N 

20 

45 

N 

20 

54 

SWUS/24 

Y 

9 

7 

Y 

11 

4 

Y 

6 

1 

N 

21 

63 

N 

26 

59 

N 

24 

69 

SWUS/36 

Y 

8 

6 

Y 

10 

4 

Y 

5 

1 

N 

22 

64 

N 

27 

59 

N 

25 

69 

In  all  models/algorithm,  regions  and  forecast  durations,  the  greatest  percentage  of  grid 
points  lie  in  the  N/N  category.  This  means  that  on  average  the  largest  portion  of  the 
region  is  actually  and  predicted  to  be  cirrus-free.  The  number  just  above  it  (fcst  Y  / 
GOES  N)  indicates  the  percentage  of  grid  points  in  which  a  model/algorithm  incorrectly 
predicts  cirrus  presence,  indicating  a  falsely  predicted  cirrus  location.  These  are  highest 
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for  MM5  for  in  the  NEUS  region.  The  lower  left  number  in  each  table  (fcst  N  /  GOES  Y) 
represents  the  percentage  of  missed  cirrus  locations  by  the  predictions.  Their  values 
are  greatest  in  the  SWUS  region  for  all  three  models/algorithm,  especially  DCF  in  the 
Winter  season.  The  fcst  Y  /  GOES  Y  are  the  correctly  predicted  cirrus  grid  points,  which 
are  consistently  more  prevalent  as  a  percentage  of  the  region  in  the  NEUS  than  the 
SWUS  across  all  seasons  and  forecast  durations. 

Tables  9-12  show  parameters  commonly  derived  from  contingency  tables  (Wilks, 
1995).  In  this  study  the  hit  rate,  false  alarm  rate  and  bias  are  used  to  analyze  the  cirrus 
location  and  spatial  extent  accuracy.  The  hit  rate  is  the  total  percent  of  grid  points  where 
model  forecast  outcome  agreed  with  the  GOES  analysis  outcome.  The  results  suggest 
that  at  any  given  grid  point  in  a  24-h  or  36-h  forecast,  the  cirrus  Y/N  forecast  has  at 
least  a  66%  chance  of  being  correct.  The  odds  are  better  in  the  NEUS  for  the  Winter 
period  and  for  the  SWUS  in  the  Spring.  MM5  and  NAM  hit  rate  differences  vary  with 
season  in  the  NEUS  but  consistently  favor  MM5  in  the  SWUS.  DCF  forecasts  are 
competitive  in  hit  rate  with  MM5  and  NAM  in  most  seasons/regions  except  SWUS  in 
Winter  and  NEUS  in  the  Spring.  Hit  rate  seems  to  decline  slightly  with  forecast  duration. 

False  alarm  rate  (FAR)  is  the  percentage  of  predicted  ice  grid  points  that  were 
incorrectly  forecasted.  High  FAR  values  indicate  the  tendency  to  over-predict  cirrus 
occurrence  or  predictions  in  areas  where  cirrus  was  not  detected.  Tables  9-12  indicate 
that  MM5  has  consistently  higher  FARs  than  the  NAM  in  ice  cloud  prediction.  MM5  and 
NAM  FAR  values  for  Fall  and  Winter  are  higher  in  the  NEUS  region  than  in  the  SWUS 
region  but  reversed  in  Spring  and  in  the  Summer  for  MM5.  The  highest  FAR  values  are 
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Table  9.  Fall  period  contingency  table  statistics  based  on  values  shown  in  Table  5.  Hit 
rate  =  %Y/Y  +  %N/N,  false  alarm  rate  =  (%  fcst  Y/GOES  N  +  %  test  Y)  X  100,  and  bias 
=  %  fcst  Y  +  %  GOES  Y. 


Region/Fcst 

Duration 

Model 

Hit  Rate  (%) 

False  Alarm 
Rate  (%) 

Bias 

NEUS/24 

MM5 

72 

41 

1.42 

NAM 

73 

31 

1.07 

NEUS/36 

MM5 

71 

42 

1.30 

NAM 

71 

33 

1.02 

SWUS/24 

MM5 

73 

31 

0.70 

NAM 

66 

27 

0.57 

SWUS/36 

MM5 

71 

37 

0.75 

NAM 

66 

25 

0.52  1 

Table  10.  Winter 

period  contingency  table  statistics  based  on  Table  6. 

Region/Fcst 

Duration 

Model 

Hit  Rate  (%) 

False  Alarm 
Rate  (%) 

Bias 

NEUS/24 

MM5 

78 

43 

1.43 

NAM 

79 

33 

1.15 

DCF 

78 

27 

0.52 

NEUS/36 

MM5 

75 

49 

1.52 

NAM 

77 

37 

1.15 

DCF 

76 

36 

0.50 

SWUS/24 

MM5 

74 

35 

0.76 

NAM 

70 

16 

0.44 

DCF 

68 

14 

0.19 

SWUS/36 

MM5 

73 

36 

0.74 

NAM 

68 

22 

0.43 

DCF 

67 

20 

0.14 

Table  11.  Spring 

period  contingency  table  statistics  based  on  Table  7. 

Region/Fcst 

Duration 

Model 

Hit  Rate  (%) 

False  Alarm 
Rate  (%) 

Bias 

NEUS/24 

MM5 

72 

33 

1.00 

NAM 

76 

19 

0.84 

DCF 

68 

34 

0.71 

NEUS/36 

MM5 

72 

35 

1.05 

NAM 

73 

24 

0.86 

DCF 

69 

34 

0.73 

SWUS/24 

MM5 

76 

42 

0.93 

NAM 

74 

28 

0.53 

DCF 

73 

6 

0.04 

SWUS/36 

MM5 

75 

44 

0.96 

NAM 

73 

32 

0.56 

DCF 

73 

0 

0.04 

45 


Table  12.  Summer  period  contingency  table  statistics  based  on  Table  8 


Region/Fcst 

Duration 

Model 

Hit  Rate  (%) 

False  Alarm 
Rate  (%) 

Bias 

NEUS/24 

MM5 

73 

39 

1.03 

NAM 

72 

29 

0.86 

DCF 

71 

37 

0.75 

NEUS/36 

MM5 

71 

41 

0.97 

NAM 

69 

31 

0.80 

DCF 

70 

38 

0.72 

SWUS/24 

MM5 

72 

44 

0.53 

NAM 

70 

27 

0.41 

DCF 

75 

14 

0.23 

SWUS/36 

MM5 

72 

43 

0.47 

NAM 

69 

29 

0.38 

DCF 

74 

17 

0.20 

due  to  over-prediction  of  cirrus  during  the  Fall  and  Winter  periods  in  the  NEUS  but  the 
result  of  mis-location  of  cirrus  predictions  in  the  SWUS  in  the  Spring  and  Summer.  This 
is  reflected  in  the  bias  scores.  Bias  is  the  ratio  of  the  total  percentage  of  ice  grid  points 
predicted  to  ice  grid  points  detected.  Tables  5  and  6  show  more  predicted  than  detected 
ice  grid  points  for  the  NEUS  in  Fall  and  Winter  for  both  MM5  and  NAM  (greater  for 
MM5),  and  more  detected  than  predicted  for  the  SWUS  (more  under-predicted  by  NAM) 
in  all  seasons  except  MM5  in  Spring.  Recall  that  just  a  single  GOES  pixel  in  a  grid  cell 
constitutes  a  GOES  ice  cloud  grid  point  for  the  purposes  of  these  statistics.  This 
suggests  that  in  the  SWUS  the  models,  especially  NAM,  misses  the  frequent  scattered, 
isolated  GOES-detected  ice  cloud  (see  for  example  Figure  8).  And  even  though  the 
SWUS  FARs  are  not  negligible  for  either  model,  the  occurrence  of  fcst  Y  /  GOES  N  in 
both  models  is  much  smaller  in  SWUS  than  in  NEUS.  DCF  has  a  lower  FAR  than  MM5 
or  NAM  except  for  during  the  Spring  and  Summer  in  the  NEUS  region.  But  it  also 
consistently  has  the  greatest  under-prediction  bias.  Less  than  !4  of  the  actual  number  of 
ice  grid  points  are  predicted  in  the  SWUS  in  the  Winter,  Spring  and  Summer  by  DCF. 
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4.5  Comparison  of  IWP,  CTH  and  Cirrus  Cover 

Though  the  emphasis  of  this  study  is  to  evaluate  the  predicted  location  of  the 
cirrus  and  its  coverage,  it  is  also  of  interest  to  consider  briefly  the  comparison  between 
predicted  and  retrieved  CTH,  IWP  and  cover.  Given  the  limitations  of  the  GOES 
retrievals  as  mentioned  in  the  first  paragraph  of  Section  3  (low  CTH  bias,  low  IWP  bias), 
the  GOES  pixel  average  within  the  model/algorithm  grid  cells  can  be  directly  compared 
with  the  model/algorithm  ice  cloud  grid  point  values  for  all  fcst  Y  /  GOES  Y  grid  points. 
Tables  13  and  14  display  the  results  of  this  comparison  at  the  Y/Y  ice  grid  points 
averaged  over  all  forecasts  for  Winter  and  Summer.  The  MM5  IWP  averages  are  2-3 
times  larger  than  the  GOES  retrieval  averages,  which  is  consistent  with  the  suspected 
low  bias  of  the  latter  values.  NAM  IWP  averages  are  consistently  an  order  of  magnitude 
smaller  than  the  MM5  IWP  averages,  and  much  smaller  than  the  suspected  small 
GOES  values.  This  is  due  to  the  partitioning  scheme  used  in  the  NAM  that  assigns  less 
than  10%  of  the  total  hydrometeor  ice  to  cloud  ice  water  mixing  ratio  and  over  90%  to 
precipitation  ice  water  mixing  ratio  (Ferrier,  personal  communication). 

Average  CTH  are  similar  for  MM5  and  NAM  in  the  Winter  season.  MM5  has 
somewhat  higher  CTH  averages  in  the  Summer  season.  Both  are  higher  than  GOES  in 
both  seasons,  consistent  with  the  low  bias  of  the  GOES  retrievals  found  by  Norquist  et 
al.  (2008).  This  difference  decreases  when  some  of  the  model  CTH  values  are  adjusted 
to  the  tropopause  height.  The  last  column  of  Tables  13  and  14  indicates  that  the 
percentage  of  ice  grid  points  with  CTH  that  exceed  the  tropopause  height  are  somewhat 
greater  in  the  MM5  forecasts  than  in  the  NAM  predictions.  Very  few  of  the  GOES  CTHs 
exceed  the  tropopause  height  in  either  season  or  region. 
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Table  13.  Winter  period  ice  water  path  (IWP),  cloud  cover  (CC,  in  parentheses)  and 
cloud  top  height  (CTH)  averaged  over  all  forecast  Y/GOES  Y  grid  cells  (all  GOES  ice 
pixels  averaged  in  cell)  over  all  forecasts.  Multiple  entries  for  GOES  represent 


comparisons  with  MM5,  N 

AM  and  DCF  respectively. 

Region/Fcst 

Duration 

Model  / 
Analysis 

Ave.  IWP  (g  m*) 
(Ave.  CC,  %) 

Ave.  CTH  (km) 

Ave.  Trop. 
Adj  CTH 

%  Trop.  Adj 
Pts. 

NEUS/24 

MM5 

60 

10.3 

9.7 

68 

NAM 

6 

10.0 

9.6 

53 

GOES 

23,  22  (88) 

9.0,  8.9,  9.4 

9.0,  8.9,  9.4 

6,  2,6 

DCF 

(52) 

13.3 

10.5 

100 

NEUS/36 

MM5 

54 

10.0 

9.6 

59 

NAM 

6 

9.9 

9.6 

52 

GOES 

23,  22  (85) 

8.9,  8.8,  9.3 

8.9,  8.8,  9.3 

7,  2,7 

DCF 

(58) 

13.5 

10.4 

100 

SWUS/24 

MM5 

41 

10.1 

9.8 

39 

NAM 

3 

10.1 

9.8 

34 

GOES 

19,  18(90) 

9.4,  9.3,  9.6 

9.3,  9.3,  9.6 

7,  5,4 

DCF 

(79) 

13.5 

10.9 

95 

SWUS/36 

MM5 

38 

10.2 

9.9 

39 

NAM 

3 

10.2 

9.9 

35 

GOES 

18,  18(88) 

9.4,  9.4,  9.5 

9.4,  9.3,  9.5 

5,  5,3 

DCF 

(76) 

13.7 

10.9 

97 

Table  14.  Same  as  Table  13  except  for  Summer  period. 


Region/Fcst 

Duration 

Model  / 
Analysis 

Ave.  IWP  (g  m'^) 
(Ave.  CC,  %) 

Ave.  CTH  (km) 

Ave.  Trop.  Adj 
CTH 

%  Trop. 
Adj  Pts. 

NEUS/24 

MM5 

28 

11.4 

11.3 

20 

NAM 

3 

11.0 

10.9 

17 

GOES 

21,  19(81) 

10.7,  10.5,  10.6 

10.6,  10.5,  10.6 

2,  4,2 

DCF 

(65) 

15.3 

13.5 

67 

NEUS/36 

MM5 

27 

11.6 

11.5 

22 

NAM 

3 

11.2 

11.1 

22 

GOES 

21,  20  (80) 

10.7,  10.6,  10.6 

10.6,  10.5,  10.5 

3,  3,3 

DCF 

(62) 

15.0 

13.5 

63 

SWUS/24 

MM5 

24 

12.2 

12.1 

10 

NAM 

3 

11.0 

10.9 

3 

GOES 

20,  19(88) 

11.1,  11.0,  11.2 

11.1,  11.0,  11.1 

0,  1,0 

DCF 

(67) 

13.4 

13.1 

14 

SWUS/36 

MM5 

23 

11.9 

11.9 

8 

NAM 

3 

11.0 

11.0 

2 

GOES 

20,  20  (85) 

11.0,  11.0,  11.1 

11.0,  11.0,  11.1 

0,  0,1 

DCF 

_ (63) _ 

13.3 

13.0 

15 

DCF  cloud  top  height  comparisons  with  co-located  GOES  retrievals  reveal  that 
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the  former  are  on  average  from  2  to  4  km  higher.  This  suggests  that  the  DCF  CTH 
values  are  probably  too  high,  especially  in  the  NEUS  in  both  seasons  and  in  the  SWUS 
in  the  Winter.  This  is  reflected  in  more  than  95%  of  the  Y/Y  DCF  CTH  values  exceeding 
the  tropopause  heights.  Grid  cell  cirrus  cover  for  compared  DCF  and  GOES  Y/Y  grid 
points  reveals  a  15-30%  under-prediction.  This  is  due  in  large  part  to  the  categorical 
nature  of  the  DCF  cloud  cover  diagnosis,  which  restricts  cloud  amounts  to  no  more  than 
90%  coverage  even  in  the  midst  of  large  cirrus  masses  (see  Figure  7).  The  great 
majority  of  GOES  grid  cell  coverages  are  overcast  in  such  environments. 

Seasonal  period  averages  of  IWP,  CTH  and  cirrus  cover  do  not  reveal  the  nature 
of  the  predictions  in  comparison  with  GOES.  Some  insight  on  this  can  be  gained  by 
examining  the  frequency  distribution  of  the  predicted  and  analyzed  values  of  these 
cirrus  properties.  This  was  done  only  for  the  fcst  Y  /  GOES  Y  points  in  order  to  allow  a 
fair  comparison  between  the  predicted  and  detected  values.  Figure  28-33  show 
frequency  distribution  plots  for  the  NEUS  region  only  for  Winter  and  Summer  periods, 
since  fcst  Y  /  GOES  Y  counts  in  the  SWUS  were  considerably  lower. 

Frequency  distribution  of  CTH  in  Figures  28,  29,  31  and  32  make  it  clear  that 
most  of  the  predicted  CTH  values  are  larger  than  the  GOES  algorithm-retrieved  CTH  at 
the  comparison  grid  points.  For  MM5  and  NAM  these  are  concentrated  at  1-3  height 
categories  larger  than  the  GOES  peak  category,  while  for  DCF  there  are  multiple 
maxima  in  the  frequency  distribution.  Again,  this  is  due  to  the  categorical  predictands  in 
the  DCF  method  that  result  in  specific  CTH  assignments.  The  cloud  cover  distribution 
for  DCF  and  GOES  shown  in  Figures  29  and  32  reinforce  the  point  made  earlier  that  the 
majority  of  grid  cells  were  analyzed  as  overcast  with  cirrus,  whereas  the  highest  cirrus 
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Figure  28.  NEUS  region  cloud  top  height  category  frequency  distribution  for  all  forecast 
Y  /  GOES  Y  grid  cells  for  the  Winter  period  from  24-h  (top)  MM5  and  (bottom)  NAM  ice 
cloud  forecasts  and  GOES  pixel  retrievals  averaged  within  the  grid  cells. 
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35 


- DCF 

- GOES 


Figure  29.  Frequency  distribution  of  (top)  cloud  top  height  and  (bottom)  cirrus  cover 
category  for  all  DCF  forecast  Y  /  GOES  Y  grid  cells  for  the  Winter  period  in  the  NEUS 
region. 
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Percent  Frequency  of  Occurrence  Percent  Frequency  of  Occurrence 


Figure  30.  Same  as  Figure  28  except  ice  water  path  category.frequency  distribution  for 
(top)  MM5  and  (bottom)  NAM  forecasts. 
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Figure  31.  NEUS  region  cloud  top  height  category  frequency  distribution  for  all  forecast 
Y  /  GOES  Y  grid  cells  for  the  Summer  period  from  24-h  (top)  MM5  and  (bottom)  NAM 
ice  cloud  forecasts  and  GOES  pixel  retrievals  averaged  within  the  grid  cells. 
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Figure  32.  Frequency  distribution  of  (top)  cloud  top  height  and  (bottom)  cirrus  cover 
category  for  all  DCF  forecast  Y  /  GOES  Y  grid  cells  for  the  Summer  period  in  the  NEUS 
region. 
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Percent  Frequency  of  Occurrence  Percent  Frequency  of  Occurrence 


Figure  33.  Same  as  Figure  31  except  ice  water  path  category. frequency  distribution  for 
(top)  MM5  and  (bottom)  NAM  forecasts. 
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cover  category  in  the  categorical  DCF  diagnosis  is  in  the  80-90%  range.  Even  so,  there 
are  a  non-negligible  number  of  DCF  ice  cloud  grid  cells  at  the  comparison  points  that 
have  much  lower  (10-20%  and  30-40%)  cover  category  values. 

IWP  frequency  distributions  based  on  fcst  Y  /  GOES  Y  grid  cells  are  shown  in 
Figures  30  and  33.  MM5  predictions  are  spread  across  the  entire  range  of  IWP 
categories,  whereas  NAM  forecasts  are  concentrated  mainly  in  the  categories  less  than 
10  g  m'2.  The  peak  category  for  the  NAM  IWP  values  is  less  than  that  of  GOES  and 
more  concentrated  in  numbers  of  categories.  MM5  actually  shows  a  minor  maxima  at 
the  highest  IWP  category,  as  indicated  in  the  MM5  map  plot  as  shown  in  Figure  6. 

5  SUMMARY  OF  CIRRUS  PREDICTION  PERFORMANCE 

Based  on  the  results  presented  in  this  paper,  the  following  summary  of  the 
characteristics  of  each  of  the  three  cirrus  forecast  techniques  is  presented. 
This  summary  can  guide  the  use  of  the  MM5,  NAM  and  DCF  cirrus  forecasts  in  support 
of  military  system  deployment  in  the  two  regions  studied  in  this  project. 

Major  Study  Findings,  Fall  Period 

•  NEUS 

-  MM5,  NAM  over-predict  cirrus  coverage 

-  MM5,  NAM  cirrus  event  timing  is  good 

-  Grid  cell  comparisons  -  model  vs.  GOES 

•  Hit  Rate  comparable  for  MM5,  NAM 

•  False  Alarm  rate  better  for  NAM  than  MM5 

•  Bias  -  MM5:  large  +,  NAM:  small  + 

•  SWUS 

-  Cirrus  coverage:  MM5  good,  NAM  small  - 

-  MM5,  NAM  cirrus  event  timing  is  moderate 

-  GRID  cell  comparisons  -  model  vs.  GOES 

•  Hit  Rate  better  for  MM5  than  NAM 

•  False  Alarm  rate  slightly  better  for  NAM  than  MM5 

•  Bias  -  MM5:  moderate  -,  NAM:  large  - 
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Major  Study  Findings,  Winter  Period 

•  NEUS 

-  MM5,  NAM  over-predict  cirrus  coverage,  DCF  under-predicts 

-  MM5,  NAM  cirrus  event  timing  is  good,  DCF  is  moderate 

-  Grid  cell  comparisons  -  model/algorithm  vs.  GOES 

•  Hit  Rate  comparable  for  MM5,  NAM,  DCF 

•  False  Alarm  Rate  worst  for  MM5,  best  for  DCF 

•  Bias  -  MM5:  large  +  ;  NAM:  small  +  ;  DCF:  large  - 

•  SWUS 

-  Cirrus  coverage:  MM5  good,  NAM  small  -  ,  DCF  large  - 

-  MM5,  NAM  cirrus  event  timing  is  moderate,  DCF  is  fair 

-  Grid  cell  comparisons  -  model/algorithm  vs.  GOES 

•  Hit  Rate  best  for  MM5;  NAM  and  DCF  are  comparable 

•  False  Alarm  Rate  worst  for  MM5,  best  for  DCF 

•  Bias  -  MM5:  small  -  ;  NAM:  large  -  ;  DCF:  huge  - 
Major  Study  Findings,  Spring  Period 

•  NEUS 

-  MM5,  NAM  over-predict  cirrus  coverage,  DCF  slightly  under-predicts 

-  Good  cirrus  event  timing  for  MM5,  NAM,  DCF  is  fair 

-  Grid  cell  comparisons  -  model/algorithm  vs.  GOES 

•  Hit  Rate  best  for  NAM,  worst  for  DCF 

•  False  Alarm  Rate  best  for  NAM,  comparable  for  MM5,  DCF 

•  Bias  -  MM5:  about  right ;  NAM:  small  -  ;  DCF:  moderate  - 

•  SWUS 

-  Coverage:  MM5  over-predicts,  NAM  slightly  under-predicts,  DCF  ~  0 

-  MM5  and  NAM  cirrus  event  timing  is  good,  DCF  virtually  no  cirrus 

-  Grid  cell  comparisons  -  model/algorithm  vs.  GOES 

•  Hit  Rate  best  for  MM5,  comparable  for  NAM,  DCF 

•  False  Alarm  Rate  better  for  NAM  than  MM5,  DCF  ~  0 

•  Bias  -  MM5:  about  right;  NAM:  large  -,  DCF:  extreme  - 
Major  Study  Findings,  Summer  Period 

•  NEUS 

-  MM5,  NAM  over-predict  cirrus  coverage,  DCF  about  right 

-  MM5,  NAM  and  DCF  cirrus  event  timing  is  good 

-  Grid  cell  comparisons  -  model/algorithm  vs.  GOES 

•  Hit  Rate  comparable  for  MM5,  NAM,  DCF 

•  False  Alarm  Rate  worst  for  MM5,  best  for  NAM 

•  Bias  -  MM5:  about  right ;  NAM:  small  -  ;  DCF:  large  - 

•  SWUS 

-  Cirrus  coverage:  MM5,  NAM  moderately  low,  DCF  extremely  low 

-  MM5,  NAM  cirrus  event  timing  is  moderate,  DCF  is  fair 

-  Grid  cell  comparisons  -  model/algorithm  vs.  GOES 

•  Hit  Rate  best  for  DCF,  worst  for  NAM 

•  False  Alarm  Rate  worst  for  MM5,  best  for  DCF 

•  Bias  -  MM5:  large  -  ;  NAM:  large  -  ;  DCF:  huge  - 
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Overall  Conclusions 

•  NAM  probably  the  best  performer  in  NEUS,  MM5  likely  best  in  SWUS 

•  DCF  suffers  from  under-prediction,  especially  in  SWUS 

•  MM5,  NAM  could  be  useful  for  next-day  regional  coverage  guidance 

•  DCF  would  benefit  from  regional  statistical  relationship  development 

•  DCF  cirrus  top  heights  should  be  reduced 

•  There  is  a  lot  of  room  for  improvement  in  operational  cirrus  prediction. 
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